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TITLE OF THE INVENTION 

BIOMECHANICS USER INTERFACE ELEMENTS FOR PEN-BASED COMPUTERS 
CROSS-REFERENCE TO RELATED APPLICATION(S) 

[0001] This application is related to and claims priority to U.S. provisional application 
entitled "User Interface Techniques For Pen-Based Computers" having serial number 
60/438,499, by Fitzmaurice et al, filed January 8, 2003; this application is related to U.S. 
application entitled "A Layer Editor System For A Pen-Based Computer" having serial number 
**(1 252. 1083), by Fitzmaurice et al, filed concurrently herewith; and this application is related to 
U.S. application entitled "A User Interface Having A Placement And Layout Suitable For Pen- 
Based Computers" having serial number **(1 252.1 087), by Fitzmaurice et al, filed concurrently 
herewith; and all of which are incorporated by reference herein. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present invention is directed to interface elements oriented and/or shaped in 
accordance with a natural motion of a user, such as conforming to an arc made by the hand 
when the elbow is pivoted and, more particularly, to a system that positions interface elements 
at a position determined by the hand position of the user, allowing the interface to pop-up where 
it is comfortable to use and lays out the interface elements according to a natural arc that is also 
easy to use. 

2. Description of the Related Art 

[0003] Pen or stylus based computers, such as tablet personal computers (PCs) and personal 
digital assistants (PDAs), are becoming popular. These computers typically allow the user to 
interact with the computer through a graphical user interface using a stylus or pen (or possibly 
even a finger). A keyboard and a mouse are typically not available. Pen-based type interaction 
can be limiting as compared to other types of interaction, such as using the mouse or the 
keyboard, alone or in combination. What is needed are improvements to the interaction 
capability of pen-based computers. 

[0004] Operating pen-based user interfaces in the tablet-PC form factor can be extremely 
awkward and uncomfortable: Repetitive actions are common and operating the user interface 
along the borders of the screen is problematic. First, the digitizer behaves poorly around the 
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borders of the screen and thus it is hard to control the cursor using the stylus. Secondly, 
positioning the hand along the digitizer edge can be uncomfortable and unstable to operate with 
precision. What is needed are solutions that employ a design that moves away from the 
digitizer edge and adheres to comfortable human motions. 

[0005] Typical interface elements, such as a slider or a menu, are rectilinearly oriented, such 
as oriented horizontally and/or vertically relative to the display. Using such interface elements 
with a pen typically requires a compound movement by the user that may include rotating at the 
shoulder to essentially slide the elbow, pivoting the elbow during the sliding motion and rotating 
the wrist at the same time. Such compound movements are more difficult than simple 
movements, such as pivoting at the elbow, and can cause fatigue in the user when receptively 
done. What is needed is an interface element that conforms to more natural simple motions by 
the user. 

SUMMARY OF THE INVENTION 

[0006] It is an aspect of the present invention to provide interface elements at a comfortable 
and convenient position for a user of a pen-based computer. 

[0007] It is another aspect of the present invention to provide interface elements that are 
oriented and shaped by natural motion arcs of the users hand, elbow and/or shoulder. 

[0008] It is a further aspect of the present invention to layout or design interface elements 
that take advantage of natural use motions. 

[0009] The above aspects can be attained by a system that uses natural user position and 
natural user motion to position and layout interface elements for a pen-based computer display. 
Graphical user interfaces, such as a slider or menu, are popped-up at a position convenient to 
the user, such as at the current position of the cursor. The interface is oriented along or shaped 
like a natural motion arc of the user. The interface arc can be a single motion arc, such as an 
elbow arc, a composite arc of an elbow arc and a wrist arc, a sequence of an elbow arc and a 
wrist arc, a compound where an elbow arc blends into a wrist arc or an arc followed by a linear 
interface. The arc is set by allowing a user to draw strokes on a display and the arc is 
determined based on the strokes. 

[0010] These together with other aspects and advantages that will be subsequently 
apparent, reside in the details of construction and operation as more fully hereinafter described 
and claimed, reference being had to the accompanying drawings forming a part hereof, wherein 
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like numerals refer to like parts throughout. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] Figure 1 illustrates right-hand elbow and wrist arcs made by moving a person hand. 

[0012] Figure 2 depicts the elbow and wrist arc for a left-handed user. 

[0013] Figure 3 depicts arcs for a standing user. 

[0014] Figure 4 depicts hard ware of the present invention. 

[0015] Figure 5 shows elbow arc drawing strokes. 

[0016] Figure 6 depicts a linear interface element oriented to a stoke arc. 

[0017] Figure 7 shows an arc shaped interface oriented to the stroke arc. 

[0018] Figure 8 shows a multiple control interface oriented to a stroke arc. 

[0019] Figure 9 shows multiple control elements staggered along an arc stroke. 

[0020] Figure 10 shows arc shaped multiple controls. 

[0021] Figures 11 A and 11 B show pop-up menus with rectilinear and arc shaped elements. 

[0022] Figure 12 shows arc influenced control zones. 

[0023] Figure 13 shows arranging interface elements along an arc with the most often used 
element under the cursor. 

[0024] Figure 14 shows an arc shaped interface that can be reoriented. 

[0025] Figure 15 shows arc shaped corner targets. 

[0026] Figure 16 depicts an arc influenced overflow palette. 

[0027] Figure 17 shows elements along an elbow arc and interface element wrist arc 
shaped. 

[0028] Figures 1 8 and 1 9 show elements along an arc with different depths and widths. 

[0029] Figure 20 shows multiple arc selection. 

[0030] Figures 21 and 22 show combinations of rectilinear menus with arc shaped menus. 

[0031] Figure 23 shows a menus with an arc shaped portion and a linear portion. 

[0032] Figure 24 depicts the operations of the present invention. 
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[0033] Figure 25 shows arc that can be combined. 

[0034] Figure 26 shows an alternative positioning of a linear arc oriented interface. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0035] The present invention provides an improved interface element for pen-based type 
computers by altering the shape of the user interface widgets, positioning them away from the 
edge of the screen and conforming them to the biomechanics of the human body. Simply put, 
the present invention provides interface widgets that are reshaped into or reoriented along "arc" 
shapes or arc like curves. For example, a slider control may not be straight or rectilinear but 
instead have a curved shape. Or the slider can be oriented along a particular portion of an arc. 
A row of button controls (such as in a toolbox) can also follow this arc shape. These interface 
elements or controls, sometimes called widgets, can be implemented using conventional bitmap 
graphics (or any other graphics drawing system such as vector graphics) with irregular hit or 
activation zones defined (such as, curved hit zones). Detection of the selection of a control (hit 
detection) can be efficiently performed by defining an internal data structure that is also a 
bitmap that matches the shape of the widget/palette. Each button or clickable item can be 
colored a unique color. Continuous items such as sliders can have a color gradient. 

[0036] The interface elements of the present invention can include or be oriented along a 
specific "idealized" or generalized element arc (or curve) shape (in degrees) for the user to 
target when interacting with the interface, although a custom user specific arc can be used. 
This element arc is based on the arc 10 of the movement of the hand, as depicted in figure 1, 
when the arm is pivoted at the elbow and/or on the arc 12 of the hand when the hand is pivoted 
at the wrist. The element arc can also be based on the arc of the shoulder and the arc of the 
fingers, as will be discussed later herein. This element arc can generally depend on the 
characteristics of each user as shoulder, arm, hand, finger lengths, etc. are somewhat different 
from person to person. The element arc varies slightly between the large gestures for an arm 
pivoting at the elbow and/or shoulder and a hand pivoting at the wrist. The arc also may vary 
depending on what device is being used (e.g., a PocketPC, TabletPC, etc. or an electronic 
whiteboard). Different orientations for the arcs 14 and 16 are typically defined for right-handed 
user (figure 1) and left-handed users (figure 2). An idealized arc can be determined by having a 
cross section of users draw arcs on a display, having a processor sample the drawn arcs, 
conventionally fit arc curves to the drawings, and then select an idealized element arc by one of 
a number of approaches. The arcs or radii of the cross section of users can be averaged to 
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obtain the idealized element arc. Alternatively, the arc radii can be fit to a normal, standard 
deviation or other type distribution and the idealized arc determined based on the distribution. 
In addition, a user could define or customize a baseline arc of comfort to be used instead of the 
idealized element arc in a similar manner by having the user draw an arc strokes on the display, 
a processor sample the drawn arcs, and conventionally fit an arc curve to the drawing strokes. 
In such an idealized or customized element arc, the radius need not be constant and can vary 
along the curve to match the particular combinations of pivots or motions by the user(s). It is 
also possible to have different "standard" arcs for different types of devices, such as a large 
radius shoulder arc for a white board, a medium radius elbow and/or wrist arc for a laptop 
computer or a digitizing tablet and a small radius finger arc for a PDA. 

[0037] The present invention also recognizes that widgets can be oriented differently 
depending on the operating position of the widget relative to the user. For example, a user 18 is 
working at an electronic whiteboard 20 (see figure 3) and the user is controlling a slider; the 
slider can be popped-up directly in front of the user working at the electronic whiteboard, e.g., in 
front of their chin. In this situation, an arc shape slider 22 can reflect roughly a downward arc, 
like a frown on a smile-face. However, if the user popped-up the slider a significant distance to 
their right, a slider 24 can be oriented sideways, like a right handed, close-parentheses ")". 

[0038] The present invention also provides for the most frequently used commands to be 
positioned along the main "dominant" arc or curve pathway. So, for example, the Alias 
SketchBook ™ application, for right-handed users, has it's main command toolbox located at the 
bottom left of the screen (the base of the dominant arc) where the hand moves from the 
command base to the main canvas region as discussed in the related application. 

[0039] Pop-up widgets, such as menus, can also conform to the arc shaped layout as 
discussed later herein. 

[0040] The present invention preferably uses a hardware platform 40 as depicted in figure 4. 
This platform 40 includes a conventional storage 42 (RAM, ROM, hard disc, floppy disc, CD, 
DVD, etc.) for storing layer editors, paint programs, data structures, data, curve fitting functions, 
etc. used by the present invention as well as the processes, bitmap graphics and bitmap hit 
zones of the invention discussed herein. The platform 40 also includes a conventional 
processor 44, typically a tablet personal computer (Tablet PC) class machine. The platform 40 
further includes a conventional pen based I/O system 46, such as a display with a pen-based 
digitizer, and optionally includes conventional I/O components, such as a keyboard, floppy disc, 
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mouse, etc. The I/O system 48 can also be the type that allows touch input by a finger and 
input by a 3D position sensing system that senses the 3D position of a finger, hand, eyes, etc. 
as well as via a mouse or other conventional input system. 

[0041] In designing an arc shaped or oriented element, the user(s) is allowed to make pen 
strokes 60 on a pen-based computer display using a conventional drawing tool as depicted in 
figure 5. This results in naturally shaped, neutral posture (no motion of fingers or wrist), arc 
shaped strokes due to the arm pivoting at the elbow (and possibly with some added curve 
influence of the hand pivoting at the wrist). The strokes are sampled and a curve is fit to these 
strokes in a conventional manner. These stroke arcs have a radius R that generally defines the 
curve of the arc. 

[0042] A linear interface element, such as a slider 62, can be oriented essentially tangent to 
the arc shaped strokes 62 (or the fit curve) at a point along the strokes, such as a mid point, as 
depicted in figure 6. Note the text "100" at the top end of the interface can be rectilinear aligned 
with the display or curved. Such an arc oriented interface element 62 is easier to use than the 
typically vertically or horizontally oriented interface element as the user can move along the 
element 62 more easily with a more to natural simple type, non-compound motion. 

[0043] An interface element, such as slider 64, can also be bent or arced to follow the arc 
shaped strokes 62 as depicted in figure 7. This curved slider 64 conforms to the element arc 
discussed above, and is much easier and more natural to use than the typical rectilinear 
interface elements. This curved slider 64 can be created by conventionally mapping the pixels 
of the bitmap of a conventional slider to a bitmap curve or vector graphic curve for the curved 
element corresponding to the arc. For example, a bit, pixel or graphic element at a particular 
length distance from a reference end of the conventional slider along a centerline of the 
conventional slider and at a width distance from the center line on one side of the centerline of 
the conventional slider are mapped to the corresponding position relative to the arc curve in the 
arc shaped slider 64. This mapping is performed for all of the graphic elements of the slider. A 
corresponding mapping occurs for the hit zone. The interaction behavior of the standard slider 
maps to the new arc shape space and orientation. Even invisible elements such as an out-of- 
range border for the standard slider maps to an arc shaped out-of-range border for the arc- 
based slider. 

[0044] A multiple control interface element, such as a menu 80, can also be oriented to the 
arc shaped strokes 60 and maintain it's rectilinear shape, as depicted in figure 8. That is, the 
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sides 81 of the controls are linear. In this case the menu 80 is also aligned with a curve 82 that 
is perpendicular to the strokes 60. This curve 82 is called an anti-arc, anti-curve or counter-arc, 
and is a curve that is oriented 90° from the elbow dominant arc of the strokes 60. This arc 
allows selection of items along this arc by slightly changing the shape of the dominant arc with a 
slight added motion added by the wrist and/or fingers, that causes the dominant arc to be 
extended above or below its natural path. Such a multiple control menu 100 can also have the 
rectilinear controls staggered along the arc shaped strokes 62 as depicted in figure 9. And a 
multiple control menu 110 can itself be arc shaped as depicted in figure 10, such that the 
controls are non-rectilinear and conform to the arc shaped strokes 62. That is, the sides 112 of 
each of the controls is arc shaped 

[0045] Pop-up toolbars 120 can also adopt the orientation of the arc as depicted in figure 
11 A. Note how the marking menu icon remains horizontally orientated for easy reading while 
the individual tool targets (hot spots) 122-128 adhere to the anti-arc extended arc orientation. 
This figure also shows the concept of a marking menu 126 as one of the tools, with visible 
commands in this case, and that has an "overflow" area 130 that, in this example, is up and to 
the right of the marking menu center along the dominant elbow type curve. As can be seen, the 
positioning of the overflow area 130 can also conform to the underlying arc paradigm. The 
shape of the overflow area 130 can take on the similar arc contour shaped palette depicted 
previously in figures 9 and 10. Figure 11 B illustrates the pop-up menu with an elbow arc 131 
shaped overflow area 132 with arc shaped controls 134 therein. This embodiment also shows 
the marking menu regions 136 oriented and aligned along a wrist shaped arc 138. 

[0046] The targets or hit zones of the controls (as well as the graphics thereof) can be 
curved as more particularly illustrated by the wedge shapes targets 140-146 of figure 12. This 
figure also shows a marking menu 148 with visible commands popping-up as a result of the 
activation of the control for zone 142. 

[0047] Figure 13 depicts a pop-up menu 160 positioned at the location of the cursor 162 
and with the selectable items 164-170 oriented along the biomechanical element arc 172. In 
this case, the pop-up 160 has four items 164-170 represented graphically as circles intersecting 
the arc 172. The pop-up 160 remains fixed until the mouse-up event occurs. On initial pop-up, 
the default item 168 is conventionally presented directly under the current position of the cursor 
162. The user can move up or down the arc 172 to select the different items 164-170. 

[0048] In figure 14 an arc-shaped command strip 190 is defined. The strip 190 is anchored 
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at a base 192 and contains user interface elements 194-198. In this case, each element 
contains an icon and a marking menu. Alternatively, other widgets and status information can 
be housed on, within or near the command strip. The command strip 190 can be manually re- 
oriented (twisted or bent) and the elements follow the new orientation. The command strip can 
behave like real-world materials and have the same dynamic properties (e.g., a metal strip with 
a given flexibility and stiffness where each item is a ball with a given physical weight). The user 
can manipulate the command strip 1 90 to enlarge or shrink it or to move it out of the way. For 
example, the strip 190 can be bent downwards, upwards, pulled-out (to expose more elements 
on the command strip, to expand individual targets on the strip, or to increase spacing between 
items on the strip). Conventional move, expansion, etc. techniques are used to re-orient, the 
strip 190. 

[0049] Figure 15 depicts another arc based interface embodiment 200. In this figure the 
bottom left corner region of the screen 202 houses a number of "targets" A-D roughly in the 
shape of triangles with arc shaped sides. Each of the regions A-D houses a different command 
that in this case is a different marking menu. The "blades of grass" each having a length or 
width W delineate the different hit zones. This interface allows for extremely wide targets when 
the user is starting roughly from the center of the display screen and is moving the cursor to the 
bottom left corner of the screen. It is known that wide targets allow for fast selection 
performance. Note that the bottom right corner with the blades of grass pointing towards the 
center could be used if the user were left-handed. 

[0050] Figure 16 is an example of a marking menu overflow region 210 positioned up and to 
the right of a circular menu region 212 (see the related application). In this example, the 
overflow region houses a variety of paint brushes, pencils and erasers. The shape of the 
palette 210 and the layout of the individual items are arranged roughly to match the 
biomechanical arc. This figure also shows the interface 212 positioned at a location accessible 
by a user when making a natural elbow pivot to swing a hand across a display toward the lower 
right corner. 

[0051] Figure 17 is an example of an interface with a mixture of elbow and wrist orientation. 
Here we see a collection of sliders 220-226 oriented along a main or elbow arc 228 and each 
slider is oriented or shaped to match the wrist shape arc. This allows the user to select a slider 
by elbow pivoting and then control the slider by wrist rotation. 

[0052] In figure 18 control items 1-3 are arranged along an elbow type arc 240 where each 
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item has a narrow but deep hit zone (a-c). The hit zone for each item also includes a region 
within the dominant arc widget. Figure 19 depicts a similar layout with wider hit zones (a-c) for 
the items 1-3. 

[0053] Figure 20 shows a sequence of arc motions 250-260 that can be used to set different 
parameter attributes. Alternatively, it can increment values (in this case, 3 increments are 
specified). 

[0054] Figures 21 and 22 illustrate combinations or rectilinear and arc shaped interface 
elements. A rectilinear menu 280 with an arc shaped sub-menu 282 is shown in figure 21 while 
an arc shaped menu 284 with rectilinear sub-menus 286 and 288 is shown in figure 22. 

[0055] Figure 23 shows a hybrid menu 290 with an arc shaped portion 292 and a rectilinear 
portion 294 allowing a combination of easy to access elements and elements that are harder to 
access such as for functions with less used consequences, such as a delete function. 

[0056] The operations of the processor in producing pop-up arc shaped or oriented interface 
(GUI) elements of the present invention are shown in figure 24. When the pen-based I/O 
system hardware detects an activation signal, such as a tap or pen down, the location of the 
cursor is determined and a determination 300 is conventionally made as to whether an arc 
based interface element has been activated by comparing the position of the pen/cursor to hit 
zone maps in storage. If the cursor/pen is in a region where a pop-up menu can pop-up, the 
system determines 302 whether the user has designated that the pop-up interface be a custom 
interface. If so, the custom arc is retrieved 304. If not, the standard or element arc is retrieved 
306. The system then positions (308) or locates the arc along, at or in association with the 
cursor/pen position. If the location also affects the orientation of the arc, such as in the 
whiteboard example previously discussed, the arc is oriented responsive to the location. 
Because the interface element pops up where specified by the user, the user can position the 
interface away from a display screen edge and at a position where it is comfortable to use. The 
interface or widget is then mapped or applied 310 along/on the arc by mapping the widget 
graphic bitmap to the arc along with a similar mapping of any control zone for the widget. The 
system then monitors further movement of the pen/cursor to perform operations associated with 
the interface. 

[0057] A non-pop-up or fixed position element can also be produced in a similar manner by 
starting with a known position for the interface, retrieving a custom (304) or standard (306) arc 
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and applying (310) the interface at the predetermined location. To reduce the interference with 
interface operation by the display edge, the fixed position arc shaped or oriented interfaces 
elements can be located to a preferred side of the display, such as on the left side for a right- 
handed user or spaced in from the display edge by the distance of a typical wrist. To 
additionally reduce the interference by the edge of the display, the fixed position elements 
preferably also "grow" towards the center of the screen and are pop-up elements. 

[0058] As previously discussed, the invention provides for the use of a combination of a 
number of arcs in setting a position of an interface and/or determining the shape of the interface 
as depicted in figure 17 the combination of the different arcs into a single composite arc. The 
elbow and wrist arcs have been primarily discussed with respect to previously presented 
figures, such as figure 1 . Figure 25 illustrates the relationships and shapes of the shoulder arc 
320, the elbow arc 322, the wrist arc 324, the finger arc 326 and the anti-arc 328 or extension 
arc. In a three dimensional input space, other arcs can also be added to or combined with the 
arcs of figure 25 to provide a natural motion interaction in such a space. For example, an arc or 
curve associated with a waist twist or waist bend can be added or combined with the curves of 
figure 25. 

[0059] Figure 6 was discussed with respect to positioning the linear arc oriented interface 
element on a tangent to the element arc. It is also possible to position the element 340 across 
the arc 342 as depicted in figure 25. Using this approach the fingers are moved inside the arc 
342 rather than outside the arc when manipulating the linear arc oriented interface element. 

[0060] It should be noted that, while the examples discussed herein show a particular 
curved arc; the invention is not limited to this style or shape of arc. Specifically, widget or 
interface elements could be a design based other shapes of arcs or curves. For example, "S" 
shaped arcs or segmented straight lines effectively creating a curved interface element can be 
provided. The curve of the interface element of the invention is also not limited to paths that 
reflect biomechanical advantages either. For example, non-arc shapes could be used to 
produce a certain visual look as opposed to a particular motion. 

[0061] The present invention has been described with respect to an interface useful for a 
tablet-stylus type personal computer. The interface and processes discussed herein are also 
useful in non-tablet type personal computers. The hit zone where a user can activate a control 
conforms to the arc of the natural motion curve has been described as coinciding with the 
interface graphic. However, the hit zone can be larger, smaller or a different shape than the 
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graphic. 

[0062] The present invention has been described with respect to pen-based systems but 
also applies to touch-screen systems and computer-vision techniques used for sensing human 
input, as well as systems that sense position in three dimensions of a finger, hand, etc. 

[0063] The present invention has been described as conforming the approach path to a Ul 
to a natural motion curve, conforming the shape of an overall Ul graphic to the natural motion 
curve and as well as shaping and positioning the control elements in conformance with the 
natural motion curve. A natural motion curve is predominantly a smooth arc; however, a natural 
motion curve can include small bumps or natural variations caused by a number of different 
things, including a motion that uses coarse motor muscles and fine motor muscles. 

[0064] The present invention has also been described with respect to positioning the 
controls along or on the natural motion arc or curve. However, such a positioning can, in some 
instances, obscure the view of the user of the control because the user's hand could cover part 
or all of the control. The controls can thus be positioned above or below the curve with above 
the curve being preferred because it is less likely to obscure the control. 

[0065] The system also includes permanent or removable storage, such as magnetic and 
optical discs, RAM, ROM, etc. on which the process and data structures of the present invention 
can be stored and distributed. The processes can also be distributed via, for example, 
downloading over a network such as the Internet. 

[0066] The many features and advantages of the invention are apparent from the detailed 
specification and, thus, it is intended by the appended claims to cover all such features and 
advantages of the invention that fall within the true spirit and scope of the invention. Further, 
since numerous modifications and changes will readily occur to those skilled in the art, it is not 
desired to limit the invention to the exact construction and operation illustrated and described, 
and accordingly all suitable modifications and equivalents may be resorted to, falling within the 
scope of the invention. 
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